A computer system typically has a natural hierarchy of storage tiers. For example, a computer may have volatile dynamic random access memory (DRAM) for fast access to currently active data, a non-volatile flash drive for moderate-speed access to recently used data that may be needed again soon, and a mechanical disk or network storage for slow access to bulk durable storage. Because the storage tiers have different latencies and capacities, some data items may be replicated or moved between various storage tiers to ensure that data is dynamically distributed to storage tiers according to actual demand.
Random access tiers, such as volatile and non-volatile memory, may contain a cache that optimizes data residency based on a policy of the cache. For example, a least recently used (LRU) cache may evict data not recently used to make room for recently used data not already cached. Caching policies decide which data items are cacheable and how many data items are cached.
However the performance of cache policies, no matter how well designed, may not be optimal. For example, the quantity and quality of data access requests may fluctuate, such that at times a cache may be under- or over-utilized.
For example if a caching policy does not match a current workload, then cache thrashing may result. For example, a stressed cache may repeatedly evict and reload a same data item that would perform better if not evicted.
Likewise, an idle cache may consume memory that would increase throughput if instead used for other purposes. However although cache performance more or less depends on the nature of the workload, cache sizes typically do not dynamically adjust to suit fluctuating workload.
Thus, workload related performance degradations may arise. For example, a database table scan may trample the current cache contents as the scan passes over copious data that far exceeds the capacity of the cache. The scanned data may be read once during the scan and then not accessed again from the cache once read into cache.
Thus a scan may not only evict frequently used data, but also further aggravates the problem by filling the cache with data that is unlikely to be accessed again while in the cache. Thus a cache whose configuration and behavior is immutable will likely suffer degraded performance under various realistic loads.
Degraded performance, such as thrashing, costs extra time (latency) to load needed data. Furthermore if the cache is non-volatile, then thrashing may cause additional wear that may reduce the life of the cache's physical medium. Thus, cache thrashing may pose a capital cost and a reliability hazard, in addition to costing latency and energy.